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SHINE in Secondaries 


Executive summary 


The project 


This is a report of a pilot programme of the SHINE in Secondaries Saturday school transition 
intervention. The programme aims to improve attainment by focusing on literacy and numeracy and 
revisiting areas where pupils are struggling through a creative curriculum approach and enrichment 
opportunities. 


The Saturday programme is run by teachers from the school for 25 weeks throughout Year 7. Schools 
are responsible for developing their own curriculum of activities to suit the abilities and needs of their 
pupils following guiding principles specified by SHINE. 


The programme is designed to run for approximately 60 pupils staffed by four qualified teachers, three 
teaching assistants and three peer mentors all recruited and employed by the host secondary school. 


This evaluation was set up as a pilot study with four schools and 613 pupils participating. This report 
is one of two studies evaluating SHINE’s Saturday programmes. Hallé SHINE on Manchester 
evaluation is an EEF-funded randomised controlled trial of the impact of the Saturday school on 
primary school pupils’ literacy, numeracy and music outcomes. 


This study had three aims: 


1. To explore the impact of SH/NE in Secondaries on literacy, numeracy and attitudes to school 
and learning outcomes 

2. To explore the organisational implications and lessons for future wider roll-out 

3. To explore the feasibility of using regression discontinuity design (RDD) as a pilot study 


The study was funded by the Education Endowment Foundation as one of 24 projects in a themed 
round on literacy catch-up at the primary-secondary transition. 


What did the pilot find? 


This pilot was designed to explore the impact of a Saturday school for secondary pupils on literacy 
and numeracy, as well as the organisational and implementation issues of the intervention. The 
feasibility of using a non-randomised evaluation approach using a regression discontinuity design 
(RDD) to evaluate the impact was also tested in this pilot. 


The evaluation showed a very small impact on literacy outcomes across all four schools using the 
Progress in English and Progress in Maths tests. These findings are limited evidence of promise of 
the intervention. A larger RDD study or randomised controlled trial would be required to investigate 
the impact of the intervention and the size of the effect (if any) with confidence. A subgroup analysis 
of children eligible for free school meals (FSM) was conducted. 


This showed no evidence of the intervention differentially benefiting pupils eligible for free school 
meals in any of the schools. These findings should be treated with caution because, due to the small 
sample size, the study was unable to detect the small effects with confidence. 


The pilot demonstrated that it was feasible to design, implement and analyse the RDD successfully; 
and the design could be used again to evaluate similar interventions where randomisation is not 
possible or desirable. 
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The process evaluation involved observations of activities, interviews and focus groups with pupils 
and teachers. However, due to the very small samples in the evaluation, all findings should be treated 
with caution. 


All teachers interviewed agreed that pupils liked the familiarity and the relaxing atmosphere that 
SHINE in Secondaries encouraged and that they learnt while being engaged in an enjoyable way. An 
important outcome reported by the teachers in two schools observed was pupils’ improved social 
skills, better relationships at all levels and increased confidence through their participation in practical 
and team-building activities. 


The five pupils who were interviewed spoke enthusiastically about their SHINE in Secondaries 
experiences. All pupils mentioned that they had made good progress in their learning while having 
fun. 


The SHINE project managers, who were also teachers in the school, were considered to be integral to 
the successful running of the project. All eight teachers interviewed agreed that SHINE in 
Secondaries gave them the opportunity to think and plan ‘outside of the box’ and they enjoyed 
teaching around a particular theme, which helped reinforce links with ‘real life’. 


It was also felt that the continuity with the school week that the SHINE in Secondaries programme 
provided for pupils in Year 7 was an excellent feature of the intervention. From our visits, it seemed 
that all SHINE in Secondaries teachers see all or at least some of the pupils during the week. This 
allowed the teaching content of the SHINE in Secondaries programme to better target pupil needs 
and also to build on the pupils’ familiarity with the teaching staff. 


How was the pilot conducted? 


This evaluation was a pilot using a regression discontinuity (RD) design. Unlike a randomised 
controlled trial, where pupils are randomly allocated to an intervention or control group, this approach 
assigns pupils to intervention and control groups according to their results on a pre-test. We began 
by looking at the range of pupil outcomes on a Progress in English test. Pupils were assigned to the 
intervention if they fell below a certain mark (the first ‘cut point’). Pupils were assigned to the control 
group if they fell above a second ‘cut point’ on the test. Pupils falling between the two cut points were 
randomly allocated to the intervention or control group, thereby creating a ‘mini’ randomised 
controlled trial (RCT) within the RDD. 


Although a randomised controlled trial design is stronger for inferring causation, for this evaluation it 
was not deemed to be suitable. The RD design guaranteed all low-attaining pupils would be offered 
the opportunity to attend SHINE in Secondaries, which was a very important consideration for 
schools. 


Uptake of places at the intervention was 73% of those who were assigned to the intervention group 
which means that 27% of those counted in the analysis did not attend the intervention. The intention- 
to-treat analysis provides average impacts of the intervention for those who were offered the 
intervention, whether or not they actually participated in the intervention. This created a ‘fuzzy’ 
regression discontinuity and reduces the robustness of the evaluation. However, average individual 
attendance of those children who accepted places was 68%. This includes any pupil that is 
considered as taking up their place by attending a minimum of one session. The average attendance 
is based on the numbers of sessions possible in each school. 


How much does it cost? 


The funding provided to each school to employ staff and run the programme was £52,000. If the 
intervention took place with 60 pupils as recommended, this is equivalent to around £870 per pupil 
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per year. Although schools in the evaluation did not pay anything towards the intervention, they 
expended considerable time, resources and effort in running SHINE in Secondaries. SHINE’s ‘Making 
it Work’ handbook explains that projects are normally up to 100% funded for the first three years while 
for Years 4 to 6 schools would be expected to fund, or find funding for, at least 50% of the costs, with 
this rising to at least 70% of costs in Years 7 to 9. 


Key conclusions 


Impact of the intervention: Very small effect sizes on the primary and secondary outcomes of 
literacy and numeracy were detected. As this was an exploratory evaluation design and the 
number of schools involved in the project was small, this made it unlikely that any impact of 
SHINE could be detected, should the true estimate of effect be small. 


2. Implementation and organisation issues: Feedback from the teachers and pupils suggested 
that the intervention was perceived to be valuable in providing an opportunity to consolidate 
literacy and numeracy skills. 


3. Feasibility of evaluation design: The purpose in piloting this design was to explore the 
feasibility of using this approach when randomly allocating pupils to an intervention group was 
not possible or desirable. The regression discontinuity design is an appropriate evaluation 
method for education interventions. However, because it is less powerful than a randomised 
controlled trial, the number of pupils involved must be two and a half times greater if a small 
effect is to be detected. 
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Introduction 


Background and rationale 


SHINE in Secondaries (a Saturday school transition programme) targets disadvantaged and 
underachieving children in Year 7 to support literacy, numeracy, the development of good social 
networks, social skills and improved self-confidence in the transition between primary and secondary 
school. The intervention was developed from the most promising elements of the primary 
supplementary school intervention SHINE on Saturday, and is focused on children who did not 
achieve Level 4 at Key Stage 2. 


The period of transition from primary school to secondary school can be difficult for many pupils, with 
a new location/building, new peers, new teachers and new expectations. There is a widely 
recognised academic ‘dip’ post transition to secondary school where progress often slows for pupils 
(McGee et al., 2003) and some make no progress at all in the first year of secondary school (Galton 
et al, 1999). Studies suggest that this dip may be linked to two separate issues. First, secondary 
teachers often underestimate the academic capabilities of Year 7 pupils and therefore do not provide 
enough challenge during the first year for pupils (Evangelou et al., 2008). Second, academic 
achievement in the first year of secondary school seems to be linked to a decreased interest in 
academic activities over the course of the year (McGee et al., 2003). The SHINE in Secondaries 
intervention aims to improve achievement of disadvantaged or underachieving pupils during their first 
year of secondary school through a programme designed to engage pupils with learning, run by 
teachers from the school. These teachers are paid by the schools to work the extra hours at the 
Saturday sessions. 


The rationale for this independent evaluation is based on the significance of the widely implemented 
SHINE on Saturday intervention, which the developers (SHINE) claim has evidence of promise from 
implementation and their evaluations using a pre-experimental design. This independent evaluation of 
the SHINE in Secondaries intervention uses a pilot rigorous design and robust methods to establish 
whether or not it has evidence of promise. The evaluation focuses on establishing an unbiased 
estimate of effect of the intervention on the short-term academic outcomes (literacy and numeracy) 
and social outcomes using a regression discontinuity design (Shadish et al., 2002). 


Intervention 


The SHINE in Secondaries programme was developed by the SHINE Trust (commonly known as 
‘SHINE’) which is designed to raise achievement in the core academic subjects through high-quality 
creative and practical learning. 


Through a competitive process, schools are granted money by SHINE to run 25 weekly Saturday 
school sessions throughout the school year for underachieving Year 7 students. Schools are 
responsible for developing their own curriculum of activities around a termly theme, to suit the abilities 
and needs of their students following guiding principles specified by SHINE. These include: 


e acore focus on literacy and numeracy; 

e opportunities to revisit areas where students are struggling and to extend into areas outside 
the weekday subjects; 

e project work which is defined as hands-on learning and opportunities for enrichment which 
enable students to apply their learning to real world contexts; 

e flexibility of the curriculum to offer trips and visits which give opportunities for children who 
lack motivation or struggle with the learning to become more engaged; and 

e an emphasis on investigative and collaborative learning and research projects which 
encourage problem solving and independent learning. 
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The programme is designed to run for around 60 pupils which ensures a high staff-to-student ratio 
with each project being staffed by four qualified teachers, three teaching assistants and three peer 
mentors, all recruited and employed by the host secondary school. The selected pupils who sign up 
to the programme are divided into three classes of up to 24 students in each ‘hub’. The Saturday 
sessions usually consist of three different lessons/activities over the day, with lunch or a snack 
provided by the project. 


Background evidence 


There have been very few research studies of the effects of Saturday school programmes specifically 
and fewer high-quality, robust studies of out-of-school programmes generally. The most relevant 
studies are of after-school or summer school programmes undertaken mainly in the United States. 


The results have been quite mixed, possibly due to the variety of programmes, the quality of delivery 
and variations in the evaluation designs. 


A meta-analysis of the effects of out-of-school-time programmes for at risk students in the US (Lauer 
et al., 2006) found small but significant positive effects on both reading and maths achievement (0.05- 
0.13 and 0.09-0.17). This review identified 35 studies (including quasi-experimental studies). 
However, only one of these included a Saturday school and this was in combination with a summer 
school; the others were after-school programmes that often involved one-to-one tutoring or summer 
schools. 


A more recent review of 30 studies of interventions that increased learning time through group 
administered programmes (either after or before school, weekend or summer schools) in the US 
found no significant effects of the programmes overall on literacy or maths achievement or on social- 
emotional skills development and only a very small significant effect on academic motivation (Hedges’ 
g of 0.04) (Kidron and Lindsay, 2014). However, significant effects on literacy and maths 
achievement were found when programmes were delivered by certified instructors (as is the case in 
the SHINE intervention) and when a traditional teaching approach was used rather than pupils 
working independently. Evidence from two studies included also indicated that programmes that used 
an experiential pedagogical approach, as is advocated by the SHINE intervention, had a significant 
and moderate effect (Hedges’ g of 0.53). 


The Sutton Trust/Education Endowment Foundation (EEF) Pupil Premium Toolkit (Higgins et al., 
2013) has also reviewed the evidence for extending school time which included after-school 
programmes. Based on meta-analyses from the US and less rigorous UK evaluation designs, they 
concluded that there were low to moderate positive effects of after-school programmes, but, as with 
the study undertaken by Lauer et al. (2006), the effects varied between studies. This suggests that 
the quality of the programme and the focus are important. 


The UK evidence from a longitudinal study by MacBeath ef al. (2001) (included in the Toolkit) 
suggested that out-of-hours study support had a small positive impact on GCSE results (a third of a 
grade in maths and three-quarters of a grade in science). Those pupils who attended study support 
also had better school attendance and a more positive attitude. 


A report by the NFER (Chamberlain et al/., 2011) commissioned by SHINE (and using tracking data 
provided by schools to SHINE on the progress of pupils attending) found that students attending the 
original primary school programme, SHINE on Saturday, who were selected to take part due to 
experiencing a measure of disadvantage, made the expected progress at Key Stage 2. Confidence, 
self-esteem and attitudes to learning were also found to have improved. However, this evaluation 
used a design without a control or comparison group and was therefore unable to derive strong 
causal inference. 
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Evaluation objectives 


1. What is the impact of SHINE in Secondaries on literacy, numeracy and attitudes to school 
and learning outcomes? 

2. What are the organisational implications and lessons for future wider roll-out? 

3. Is the use of regression discontinuity design (RDD) in a pilot study feasible? 


Project team 


The evaluation team 


The evaluation team, comprising staff from Durham University and the University of York, was 
responsible for the design, conduct, analysis and reporting of the independent evaluation including 
data collection. 


Durham University, School of Education 
Professor Carole Torgerson—Principal Investigator 
Durham University, CEM 

Dr Andy Wiggins—CEM Project Manager 

Dr Dimitra Kokotsaki—Process Evaluator 
Victoria Menzies—Project Researcher 

Kirsty Younger—Project Administrator 

Clare Collyer—Project Administrator 

University of York, York Trials Unit 

Professor Catherine Hewitt—Senior Statistician 
The implementation team 


SHINE was responsible for recruiting schools and awarding grants to the schools involved to run the 
intervention. Schools were responsible for informing parents and pupils. SHINE was responsible for 
intervention development, training and support to staff running the programme in schools and for 
providing data on level of attendance of students at the intervention. 


SHINE 

Paul Carbury—SHINE Chief Executive 

Fiona Spellman—SHINE Senior Programme Manager 
Caroline Davies—SHINE Senior Programme Manager 


Clare Gilhooly—SHINE Interim Programme Manager 
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Ethical review 


Ethics approval was granted on 30 July 2013 by the Durham University School of Education Ethics 
committee. 


School headteachers received an information sheet detailing the evaluation and signed a 
memorandum of understanding agreeing to the research requirements and use of pupil data. Copies 
of the information sheet and consent forms can be found in Appendices 2-5. 


Schools involved were asked to send out a letter to all parents of children in Year 7 outlining the 
purpose of the study and allowing them to opt out of their child’s data being used in the research. 
Parents were required to contact either the research team or the school to opt out of the research. 
This did not affect whether the child could attend the intervention. For the process evaluation, opt-in 
parental consent was obtained for pupil focus groups. 
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Methodology 


Design 


The SHINE in Secondaries programme has been piloted in two schools in London; however, it has 
not previously been subject to a rigorous evaluation. The design for this research project is a pilot 
quasi-experiment using regression discontinuity design. This design was agreed owing to concerns 
by the developer that random allocation could mean that some underachieving pupils who could 
potentially benefit from the intervention would not be selected. Originally the EEF intended to fund a 
randomised controlled trial design (RCT) (the evaluators’ preferred design) and the project grant for 
delivery of the intervention to SHINE was allocated on this basis. After discussions between the 
evaluator and developer and EEF it was agreed that the RCT design would have meant that EEF 
would not be testing SHINE’s usual model. This was because, in using an RCT design, there would 
have been insufficient underachieving pupils within each school to have both a treatment and a 
comparison group, without raising considerably the threshold for being offered the intervention. 
Therefore EEF asked the evaluators to explore a variant of the RDD and it was agreed that an RD 
design would be piloted. EEF had never funded such a design before and it was agreed it would be 
useful to understand the advantages and disadvantages of using it, as well as estimating the effect of 
the intervention. As this was an exploratory evaluation design and the number of schools involved in 
the project was small, the study was established as a pilot, meaning that the minimum detectable 
effect size was relatively large (0.5), making it unlikely that the impact of SHINE could be detected, 
should the true estimate of effect be small. Despite the low power of the study it was considered to 
be worthwhile undertaking the evaluation as it would enable a point estimate of effect, with 
corresponding 95% confidence intervals, to be obtained which could be used to inform a decision as 
to whether or not the intervention should be widely implemented; or, if the pilot provided evidence for 
promise, then a larger study could be commissioned. This was approved by all in the protocol. 


The choice of a regression discontinuity design allows the pupils who are most at need to attend the 
intervention. To the best of our knowledge this design has not been used previously for this type of 
evaluation. However, it should be noted that due to the small scale of the trial, with only four 
intervention projects, it was not sufficiently powered to detect a modest, but educationally significant, 
effect although it was able to provide an unbiased estimate of effect and it could have found a large 
statistically significant effect should one have existed. Unbiased estimates of effect are reliant on 
accurate model specification. In order to ensure this a number of different models were explored (See 
below). 


This evaluation used the quasi-experimental design of regression discontinuity (RD) with two cut 
points (Shadish et al., 2002) and tie-breaker randomisation. Assignment of participants to the 
intervention (SHINE in Secondaries) was on the basis of falling be/ow the first cut point on the pre-test 
(assignment variable); assignment to the control group was on the basis of falling above the second 
cut point on the pre-test (assignment variable). This meant that pupils below the first cut point were 
eligible to receive the intervention and pupils falling above the second cut point were not eligible to 
receive the intervention. Pupils falling between the two cut points were randomly allocated to the 
intervention or control group on a one-to-one allocation. In the original design in the protocol, those 
falling between the two cut points would have been allocated to a wait list in a random order. 
However, following advice from experts on this methodology we were advised that the design would 
be stronger if the random allocation between cut points was done on a one-to-one ratio instead of the 
unpredictable waiting list where none, some, or all participants might be assigned to the intervention 
group. The use of the tie-breaker randomisation RD design was utilised to enable normal practice in 
the implementation and delivery of the intervention to be retained as requested by SHINE and as 
agreed by all parties when the protocol was approved. In normal practice, if places are not filled due 
to lack of recruitment, additional children are offered the intervention to fill up the remaining places. 
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All children below the first cut point were eligible for the intervention and were immediately offered it. 
All children above the first cut point were not immediately eligible. Because of the element of chance 
due to the error value in the test the children scoring just above the first cut point formed a waitlist of 
children who could benefit from the intervention. Having the second cut point allowed a small number 
of children between both cut points to become eligible for the intervention should places become 
available, due to low recruitment. Because the children between the two cut points were not initially 
eligible there were no ethical concerns associated with only offering half of those in the waitlist the 
intervention. 


The pre-test (assignment variable) was scores on Progress in English 11 literacy assessment. The 
cut points were decided after pupils completed the pre-test and were based on around 70 participants 
being allocated to the intervention below the first cut point and around an additional 10 pupils being 
randomly allocated from between the two cut points. This was in line with the capacity of the 
intervention and expected uptake so that between 60 and 70 pupils would attend the intervention in 
each school. This was a change to the original design where it was previously stated that we would 
pre-specify the cut points before the assessment. In practice this was not possible as it would have 
been too difficult to ensure that enough, but not too many, pupils in each school would be eligible for 
the intervention. Due to pragmatic considerations of the size of the schools and to strengthen the 
design it was necessary to have different cut points for the large and small schools. For the two 
larger schools the first cut point was set at a score of 17 and the second cut point at a score of 21. 
For the smaller schools the first cut point was a score of 22 and the second cut point was a score of 
25. This had implications for the approach to the analysis which is discussed below. 


Although a randomised controlled trial design is stronger for inferring causation, for this evaluation it 
was not deemed suitable by the funder and the developer. The reasons for this are two-fold. First, 
the grant application process was complex: as schools that applied to host the intervention were 
selected on the basis of pupil eligibility and school capacity to successfully deliver the intervention, it 
was not possible to randomise at the school level. Second, randomisation within the school would not 
have been possible as the intervention was designed and resourced to support around 60 participants 
who were underachieving and there would not have been enough underachieving pupils in the school 
for this to work. The developer was very keen for the programme to be run at capacity to ensure that 
the grant money given to schools was benefiting the maximum possible number of children. If the 
intervention had been run for a smaller number of pupils the staffing ratios would have been higher 
and the intervention would have been different from the intervention that was intended, making it 
difficult to evaluate. The RD design was also thought to be more acceptable to the schools in that all 
low-attaining pupils would be offered the opportunity to attend. The RD design allows causation to be 
inferred, with the inferences strongest around the cut points. The randomisation between the two cut 
points strengthens the design by forming a mini randomised trial at the point where inference is 
strongest. 


Although, unlike other non-randomised designs, the RD design allows strong causal inference to be 
derived, there are some limitations to the approach. First, it is less powerful than an RCT, as to get 
the same power we would need approximately 3—4 times more pupils than would an RCT (Schochet, 
2008). Second, there needs to be a clean break at the two intervention cut-offs. If some pupils below 
the first cut point do not get the intervention, or some pupils above the second cut point do get it, then 
this introduces a ‘fuzzy’ RD design and inference and power are weakened (Shadish et al., 2002). 
RD design has rarely been used prospectively in education or other research. Usually the design is 
implemented retrospectively by researchers taking advantage of a natural experiment. To undertake 
the design prospectively, as described in this report, is both novel and more robust than the usual 
retrospective approach. This evaluation is therefore piloting the design in a small study of four 
schools. 


Education Endowment Foundation 11 


SHINE in Secondaries 


Eligibility 
Schools 


Schools were eligible to participate if they were secondary schools in a targeted area awarded a grant 
by the SHINE Trust, to run the SHINE in Secondaries programme. London and Manchester were 
selected by SHINE as targeted areas where there was need and demand for the programme. The 
criteria to be awarded a grant was that schools were expecting to have at least 80 children coming 
into Year 7 who had not achieved a National Curriculum Level 4 in English and they served areas of 
disadvantage as measured by a higher than average proportion of children eligible for free school 
meals. They also had to have the ability, capacity and commitment to run the SHINE in Secondaries 
programme as judged by the SHINE Trust (information used in making this decision is described in 
the recruitment section). Two awards in each targeted area were made. 


Pupils 


All Year 7 pupils who completed the pre-test in each participating school were eligible to participate in 
the study unless parents chose to opt out of their child’s data being used or the school considered 
that a pupil had special educational needs so the intervention would not be accessible to them. If 
parents opted out, pupils could still participate in the intervention but they would not be included in the 
analysis. Assignment to the intervention or control group was done using a literacy test as described 
in the design section above: those scoring below a first cut point were invited to attend the 
intervention; those scoring above a second cut point were in the control group; and those scoring 
between the two cut points were randomly allocated to the intervention or control group. 


Consent to take part in the research was obtained through head teachers in schools at the beginning 
of the project. Schools were asked to send opt-out consent forms for the use of pupil data in the 
research and the schools obtained opt-in consent from parents for the children to attend the 
intervention. (See appendices.) 


Intervention 


Schools were asked to invite all children who scored below the first cut point on the pre-test and those 
who were randomly allocated to the intervention (having scored between the first and second cut 
points) to attend the SHINE in Secondaries programme. The research design used intention-to-treat 
analysis and therefore any pupil who was eligible to attend the intervention was analysed as part of 
the intervention group whether they attended the programme or not. 


The SHINE in Secondaries Programme 


The SHINE in Secondaries programme is a grant-funded Saturday school programme, run by a 
secondary school for pupils in Year 7. In this research there were four SHINE in Secondaries projects 
running, each in a different school. The projects were designed to run with between 60 and 70 pupils 
who were underachieving (based on their pre-test literacy scores). For this research project the 
programme was co-funded by SHINE (Support and Help IN Education) and the EEF (Education 
Endowment Foundation). 


Three of the four projects ran for 25 Saturday sessions for four hours each week while the other 
project ran for 21 Saturday sessions of five hours each. All projects ran from November 2013 to 
June/July 2014. Each school was provided with the SHINE in Secondaries ‘Making it Work’ 
handbook to guide them in setting up and running the intervention. This included guidance on the 
recruitment of staff required for the project, premises management, finance, selecting eligible 
students, encouraging attendance, SHINE’s monitoring and evaluation arrangements, and guidance 
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on the curriculum. Staff in schools were also able to contact SHINE for advice on any issues with 
running the intervention. 


A core part of the intervention was the high staff-to-pupil ratio. Minimum requirements were that each 
project was staffed by four qualified teachers—the project manager and three tutors, three qualified 
teaching assistants and three peer mentors (older pupils from the school who were compensated for 
their time in gift vouchers). All were recruited and employed by the host school, and all were paid 
additionally for their work on a Saturday. SHINE gave details of how much of the grant should be 
paid to staff (along with other expectations of how the grant was to be spent). Schools reported to 
SHINE with details on how the grant was being spent. One of the projects recruited two additional 
tutors to lead in English and maths and they were present each week. 


Pupils in each project were divided into three classes of up to 24 students. The Saturday sessions 
usually consisted of three different lessons/activities over the day with lunch or a snack provided by 
the project, as well as break time and some time at the beginning and end of the day when the whole 
group was together. 


Each project was responsible for developing their own curriculum to suit the needs of their students 
which was based on the guiding principles specified by SHINE and included (from the SHINE in 
Secondaries ‘Making it Work’ handbook): 


e acore focus on literacy and numeracy, based on a termly theme chosen by the school; 

e opportunities to revisit areas where students are struggling and to extend into areas outside of 
the weekday subjects; 

e project work, hands-on learning and opportunities for enrichment which enable students to 
apply their learning to real world contexts; 

e a thematic approach which offers opportunities to explore new areas beyond the national 
curriculum; 

e flexibility of the curriculum to do project work, trips and visits which offer opportunities for 
children who are demotivated or struggling with learning to re-engage; and 

e an emphasis on investigative and collaborative learning and research projects which 
encourage problem solving and independent learning. 


Projects were expected to include appropriate educational visits and were given a budget to 
incorporate these. These were expected to offer students opportunities beyond their current 
experience and to connect to the termly theme and a specific subject area. 


Each project manager was expected to complete a medium-term plan for each term which 
demonstrated how learning linked up between Saturdays and how enrichment opportunities could be 
integrated. This plan was shared with the SHINE programme manager and with the tutors at that 
project (an example plan can be found in Appendix 1). Tutors were then responsible for planning 
individual lessons. The project manager worked with tutors to ensure lessons were differentiated 
effectively and was responsible for quality assurance. The SHINE programme manager maintained 
regular contact with the project managers throughout the year and visited each project two or three 
times. 


Originally there was funding for schools to run a summer school for the Year 8 mentors; however, due 
to grants being made in July there was no time to set this up. Schools were therefore free to spend 
the money allocated to this in other ways to support the intervention. This was spent on residential 
trips for the children involved (in two schools), for additional enrichment activities, or for additional 
staff (as above) or to pay staff slightly higher rates to attract and retain the best staff. 
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Control condition 


Due to the regression discontinuity design, the control condition was students who had performed 
above the second cut point in the Progress in English 11 pre-test and children who performed 
between the first and second cut points and were randomised to the control condition. Schools were 
asked not to invite these pupils to attend the intervention and these pupils therefore received 
‘business as usual’. However, based on attendance data provided by SHINE we note that four control 
pupils did attend the intervention (two pupils in one school and one pupil in each of two other 
schools). The data for these pupils were still included in analysis as part of the control group as per 
the intention to treat analysis. The ‘business as usual’ control condition was not being invited to 
attend SHINE on Secondaries. 


Outcomes 
Primary outcome measure 


The total raw score on the Progress in English 12 (PIE 12) GL assessment was the primary outcome 
measure. This test is a general measure of literacy skill; the exercises included are divided between 
editing exercises and reading exercises. In the editing tasks pupils are asked to correct errors 
present in a text including spelling, grammatical and stylistic errors. The reading tasks assess text 
comprehension and understanding of language in context. This assessment was selected due to the 
intervention having a focus on core literacy and the Progress in English 12 being a valid and reliable 
assessment aimed at students in this age group and being on the EEF recommended list. 


Secondary outcome measures 


1. The total raw score on the Progress in Maths 12 (PIM 12) GL assessment was used as a 
secondary outcome as the intervention aims to improve numeracy achievement through a core focus 
on numeracy. This test is split into two sections, calculator and non-calculator, and lasts one hour in 
total. The questions addressed Levels 3 to 6 of the National Curriculum and covered number, 
algebra, shape and space, measures, data handling and probability. This test again provided a 
standardised measure of maths achievement appropriate to the age and curriculum stage of the 
students involved. 


2. Engagement with school and learning was measured by an online survey developed by the 
research team from a review of the engagement literature. The development of this survey was still in 
its piloting phase with scale validation and reliability being gathered from this project as well as the 
pilot year of another project. The survey aimed to capture pupil engagement with school including: 


e identity with schoo! (16 items including: ‘I feel like part of the school’, ‘| make friends easily at 
school’, ‘Teachers in this school care about pupils’); 

e motivations at school (13 items including: ‘I like work that makes me think’, ‘I enjoy learning 
new things’, ‘What | learn at school will be important in my future’); 

e beliefs about ability (6 items including: ‘| am confident | will succeed in school’, ‘I think I'll get 
good marks in English’); 

e active participation in learning (15 items including: ‘I get involved in class discussions’, ‘I ask 
questions to help me learn’); and 

e behaviour in completing school work (7 items including: ‘| plan my work before | start’, ‘I try to 
understand how different ideas fit together). 


To create the measure we started with reviewing the literature to identify how ‘engagement’ is 
conceptualised and to identify existing measures of engagement. Items from as many engagement 
self-report measures as could be sourced were collated into a database and categorised. The 
themes most relevant to the aims of the interventions being evaluated (which included improving 
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relationships with peers and teachers, increasing passion for learning, linking school with real life, and 
improving confidence and learning skills) were selected from the review. Items from these themes 
were adapted to suit the UK context and to be of appropriate age. Items were not specific to 
individual lessons but to school and classes generally. For most items, pupil responses were given 
on a four-point Likert scale response. All items were positively worded. Initial analysis of the data 
indicated that reliability was best for the whole scale (rather than looking at subscales) and it was 
decided therefore to use the total score as the outcome measure, where a more positive score 
indicated greater engagement with school. Engagement with school and with learning was expected 
to improve based on the theory of change model specified in the SHINE in Secondaries handbook, 
with pupils getting to Know teachers and other pupils in school better and with the relevance to school 
work being highlighted through linking learning to ‘real life’. 


Delivery of outcome measures 


All Year 7 pupils in the participating secondary schools undertook the Progress in English 12 and 
Progress in Maths 12 tests in the last week of June or first week of July 2014. The tests were 
delivered under exam conditions by a researcher from the project with invigilation support from 
teachers in the schools. A few pupils with special educational needs who teachers felt would not be 
able to sit the test in the exam hall completed the tests with a teacher or learning assistant in a 
classroom setting at the same time as the other pupils. 


Tests were marked by temporary staff employed by the research team who were blind to allocation 
(i.e. markers did not know whether test papers were from intervention or control group participants). 


The Engagement with School and Learning survey was delivered and marked automatically online. 
Schools were asked to get pupils to access a given link and complete the survey during June or July 
2014. 


Sample size 


Ideally, the evaluators would decide what constituted an educationally important difference and then 
select a sample size with adequate power (either 80% or 90%) to detect such a difference. However, 
in this instance the evaluators could not determine an optimum sample size. The sample size was 
determined by the number of SHINE in Secondaries projects funded to run in the evaluation. The 
sample included all Year 7 pupils eligible in the four schools funded. This was a pilot evaluation 
owing to the untested nature of the research design, and a sample size of four schools including 613 
children was considered adequate to pilot the procedural aspects of the design. The sample size of 
four schools was determined by the funder. The sample size for an RDD is determined by the desired 
minimum detectable effect size, significance level, power and RDD model R-squared (Lee and Munk, 
2008). Given that the sample size in each school was between 127 and 176 and assuming 80% 
power, 5% significance level and RDD model R-squared of 0.5 then the MDES would be 0.5. 


Randomisation 


The random allocation to intervention or control for students who fell between the two cut points was 
done separately for each school using pre-test and gender as stratification factors in a permuted block 
randomisation scheme. This was carried out blind by the research team’s statistician. 


Analysis 


The primary analysis was by intention to treat, i.e. those children selected to be offered the 
intervention were analysed in the intervention group, whether or not they accepted the offer of the 
intervention and received the intervention. The median pre-test measure of the randomised group for 
each school was subtracted from all scores within the school (i.e. centred). By subtracting the median 
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of the randomised group, the regression discontinuity gap is shifted to the intercept reducing the 
analysis to a classical regression model. 


Parametric estimation using all of the available data was used to model the outcome and different 
functional forms for the rating variable were explored using a regression framework. An alternative 
would have been to limit the analysis to data that lie within a window (bandwidth) but given the small 
sample size this was not deemed appropriate (Jacob et a/., 2012). To ensure the correct functional 
form was being used, a number of different functional forms were explored and models were 
compared appropriately. For each school the following models were explored: linear models, 
quadratic models, cubic models, linear models with a treatment interaction, quadratic models with a 
treatment interaction and cubic models with a treatment interaction. Models were compared using the 
Akaike’s information criterion (lower values preferred) and likelihood ratio tests were undertaken to 
compare the models. Residuals of the model were inspected as well as the visual appropriateness of 
the fit of the model to the data. Sensitivity analyses were undertaken to explore the effect of 
fuzziness around cut points by excluding pupils on waitlist from the analyses; we note that this 
approach has less statistical power than the primary analysis (Trochim, 1984). The randomisation 
adds weight to any fuzziness that might occur due to movement around the cut point because it is 
random. Robustness checks were also conducted to add confidence to the choice of model. The 
outermost data points have substantial influence on the estimation of the model, therefore it is 
important to explore how sensitive the selected functional form is to the exclusion of these outmost 
points. Hence the same models were re-estimated after sequentially dropping the outermost 1%, 5% 
and 10% of data points with the highest and lowest rating values, which are the values recommended. 


As different cut points were used for the two larger schools and the two smaller schools, the analyses 
were undertaken separately for each school and the results combined using a meta-analysis. 
Undertaking the analysis at the individual school level was appropriate here to ensure the clustering 
and the use of different cut points within schools was adequately captured in the analysis. All 
analyses were undertaken in Stata version 13. A pre-specified subgroup analysis was conducted. 
The effect of the intervention on pupils eligible for FSM was assessed through the repetition of the 
primary analysis for this subgroup and through inclusion of an interaction between FSM status and 
group for each school. 


An analysis of the pupils who were on waitlist, those between the two cut-offs, was undertaken 
separately. Linear regression was used to compare the two groups with adjustments made for the 
potential clustering within schools using the Huber-White sandwich estimator (robust standard errors). 
The outcome modelled was the PIE 12 score and the model included the PIE 11 score (pre-test) and 
group allocation. This was an exploratory analysis due to the small sample size. 


Process evaluation methodology 


A ‘light touch’ process evaluation was conducted using a cross-sectional design with interviews, focus 
groups and site visits to assess the organisational and management issues with the primary aim of 
informing longer-term, wider roll-out plans. The process evaluation was light touch owing to resource 
constraints within the funding for the evaluation as part of the EEF transitions round, thus limiting the 
number of visits (observations, focus groups) that could be undertaken. The process evaluation was 
limited to a visit by an evaluation team researcher to a project in London and a project in Manchester. 
During these visits the researcher observed three lessons in each project, ran focus groups with 
teachers in both projects and ran a focus group with students in one project. SHINE was asked to 
suggest the two projects (one in each area) to be visited for the process evaluation in order that the 
evaluator could not be seen as visiting only projects that were not going well. 


An interview with the SHINE programme manager and SHINE chief executive was also conducted by 
two members of the evaluation team during September 2014 to discuss impact and implementation. 
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Impact evaluation 


Timeline 

EA Activity 

February to June 2013 Recruitment period for schools to apply to SHINE 
for funding 

18 June 2013 

July 2013 Meetings with successful schools to explain 


evaluation and intervention requirements, and get 
consent 


September 2013 
September 2013 
October/November 2013 — June/July 2014 
June/July 2014 
August/September 2014 


October 2014 Data cleaning and application to NPD 


November/December 2014 Report writing 
Participants 


SHINE was responsible for the recruitment of schools to the trial. Expression of interest forms were 
sent out to targeted schools in areas of economic deprivation in London and Manchester. Queries 
were received from ten schools and six of these schools were shortlisted to make full applications. 
Four grants were awarded based on the need and demand in the local community and the capacity of 
the host school to deliver the intervention. The decision process to award grants included a meeting 
with the headteacher during a visit to the school, a review of Ofsted reports, taking up references 
locally and the information included in the written application. 


The four schools that were recruited to the project were urban schools in the Manchester or London 
area. One school was a boys’ school, three schools were academies and one was a community 
school. School characteristics are included in Table 1 below. 


Table 1: School characteristics (N=4, figures from 2013 obtained from Ofsted Dashboard, 
accessed December 2014) 
Characteristic | Range 


% FSM in school 54-73 
% SEN in school 


Number of pupils in school 546-1029 
% girls 
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Figure 1: Flow diagram of participants through the study 


Schools recruited to trial 


(N=4 schools) 


Excluded (n=2 pupils) 
Intervention not suitable 


Completed pre-test (N=4 
schools, n=613 pupils) 
School 1 (n=147) 
School 2 (n=163) 
School 3 (n=176) 
School 4 (n=127) 


Scored between cut points (n=82) 
School 1 (n=22) 
School 2 (n=21) 
School 3 (n=21) 
School 4 (n=18) 


Scored at or below first 


cut point — Intervention 
(n=284) intervention (n=4i) Scored above second cut 
. - Control (n=41) point — Control (n=247) 
School 1 (n=72) School 1 (n=11) 
School 1 (n=11) School 1 (n=53) 
School 2 (n=65) School 2 (n=11) 
School 2 (n=10) School 2 (n=77) 
School 3 (n=80) School 3 (n=10) 
School 4 (n=67) School 4 (n=9) School 3 (n=11) School 3 (n=75) 

. ~ School 4 (n=9) School 4 (n=42) 
eee ee Received control (n=41) Received control (n=243) 
; Sakae : Baie ae Did not receive control Did not receive control 

Did not accept invitation Did not accept invitation (n=0) (n=4) 
to attend intervention to attend intervention ~ 7 
(n=75) (n=13) 


Control Group 
Lost to follow up (n=22) 
School 1 (n=1) 
School 2 (n=6) 
School 3 (n=8) 
School 4 (n=7) 


Intervention Group 
Lost to follow up (n=42) 
School 1 (n=8) 
School 2 (n=12) 
School 3 (n=12) 
School 4 (n=10) 


Intervention Group Control Group 
Analysed (n=283) Analysed (n=266) 
School 1 (n=75) School 1 (n=63) 
School 2(n=64) School 2(n=81) 
School 3(n=78) School 3(n=78) 
School 4(n=66) School 4(n=44) 
Excluded from analysis (n=0) Excluded from analysis (n=0) 
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Pupil characteristics 


Table 2: Baseline pupil characteristics for intervention and control groups 
Means (and standard deviations) Naltavclaliceya xe} al ixe)| 

(N= 725) (NEZst5)) 
Pre-test PIE 11 raw score 
Missing 0 0 


Age at pre-test in months 
Missing pe OL 
IDACI (deprivation index 
Missing [8 ff. 4 4 
Claimed Free School Meals at any point in 
last 6 years (percentage) 

Missing ee ae 
Percentage female 
Missing POC lO 


Outcomes and analysis 


KS2 results 
Missing 


Figure 2 graphically displays pre-test scores (assignment variable: Progress in English 11 literacy 
assessment) against post-test scores (Progress in English 12 literacy assessment) by school. 


Figure 2: Summary of pre-test scores against primary outcome (PIE) by school 
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From the graphical displays of the data there does not appear to be any large discernible discontinuity 
in the outcome at the cut points and certainly no consistent pattern demonstrated across schools. 
From Figure 2, it may appear that school 2 has a different relationship between the pre- and post-test 
scores from the other three schools. However, when the regression lines are fitted to the data (see 
Figure 3) the selected model appears to be appropriate and is in line with the other three schools. 
There did not appear to be any evidence of jumps at other points across any school. While it is 
important to visually explore the data for any evidence of a discontinuity gap, it is also appropriate to 
explore the relationship statistically and we did this using a parametric approach with regression 
techniques. The results of the primary analysis are presented in Table 3 and Figure 2. To ensure the 
correct functional form was being used a number of different possibilities were explored and models 
were compared appropriately. The above graphs were also inspected to check that the specified 
functional form was intuitive. Figure 3 displays the best fitting models for each school. 


Figure 3: Summary of pre-test scores against primary outcome by school with fitted 
regression lines 
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Pooled effect sizes for the primary analysis resulted in a very small effect size between the 
intervention and control at the discontinuity in the outcome at the cut point (pooled effect size 0.09, 
95% Cl -0.08 to 0.26). It is worth noting that three of the four schools had positive effect sizes 
(ranging from 0.10 to 0.19), while one school had a negative effect size (-0.12). The results of the 
sensitivity analyses of the primary outcome are also presented in Table 3. 
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Table 3: Results of the primary analysis and sensitivity analyses of the primary outcome 


School1 | School 2 School 3 School 4 Pooled 
Primary analysis 
Estimate 1.70 1.32 1.40 -0.88 0.75 
(95% Cl) (-2.03 to (-2.68 to (-2.21 to (-4.16 to (-1.06 to 
5.42) 5,33) 5.00) 2.39) 2.57) 
Effect size (95% Cl) 0.19 0.10 0.15 -0.12 0.09 
(-0.15 to (-0.23 to (-0.16 to (-0.50 to (-0.08 to 
0.52) 0.43) 0.47) 0.26) 0.26) 
Excluding randomised 
group 
Estimate 2.40 6.48 0.59 -0.90 1.88 
(95% Cl) (-2.68 to (1.24 to (-3.88 to (-5.12 to (-1.16 to 
7.49) 11.72 5.06 3.32 4.91 
Effect size 0.23 0.57 0.06 -0.11 0.19 
(95% Cl) (-0.15 to (0.21 to (-0.28 to (-0.52 to (-0.09 to 
0.60) 0.93 0.39 0.30 0.47 
Robustness checks 
Excluding outmost 1% 
Estimate 1.29 1.21 1.92 -0.60 0.86 
(95% Cl) (-2.48 to (-2.89 to (-1.68 to (-3.93 to (-0.97 to 
5.05) 5,31) 5,52) 2.73) 2.70) 
Effect size 0.14 0.09 0.21 -0.09 0.10 
(95% Cl) (-0.20 to (-0.24 to (-0.11 to (-0.47 to (-0.07 to 
0.48) 0.42) 0.52) 0.30) 0.28) 
| Excluding outmost 5% 
Estimate 1.28 
(95% Cl) (-2.63 to 
5,19) 
Effect size 0.14 
(95% Cl) (-0.20 to 
0.48) 
Excluding outmost 10% 
Estimate 1.80 
(95% Cl) (-2.41 to 
6.02) 
Effect size 0.19 
(95% Cl) (-0.16 to 
0.54) 


The results of the sensitivity analysis demonstrate inconsistent results when excluding those 
randomised and also when the outmost 1%, 5% and 10% were excluded from the model highlighting 
concerns about the robustness of the findings and functional form selected. Nevertheless, pooled 
results all demonstrated effect sizes that were very small and in line with the magnitude found for the 
primary analysis. The results of the secondary analyses are presented in Table 4. One school did 
not complete the Engagement with School and Learning survey despite considerable prompting from 
the research team. Issues with access to the computer suite for the year group and other activities at 
the end of term were the reasons given. There was also a rather poor response rate from one of the 


other schools with only around 20% of their pupils completing the survey. 
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Table 4: Summary of the pooled results of the secondary analyses 


Summary Estimate (95% Cl) Effect size (95% Cl) | 
Progress in maths 
Main 0.82 (-1.42 to 3.06) 0.08 (-0.11 to 0.26) 
Excluding randomised group 1.64 (-0.93 to 4.22) 0.13 (-0.06 to 0.31) 
Engagement score 
Main -1.22 (-28.09 to 25.66 -0.003 (-0.73 to 0.73 
Excluding randomised group 0.63 (-24.01 to 25.27) 0.01 (-0.59 to 0.61) 


Pooled effect sizes for the secondary analyses resulted in very small effect sizes between the 
intervention and control at the discontinuity in the outcome at the cut point (PIM12 pooled effect size 
0.08, 95% Cl -0.11 to 0.26; Engagement pooled effect size -0.003, 95% Cl -0.73 to 0.73). The 
planned sensitivity analysis using mean Progress in English score (pre-test) was not undertaken as 
the means and medians were similar within each school (Table 5). 


Table 5: Summary statistics of Progress in English scores by school 
NV Tey-Ta mt &=12)) Median (25™, 75") 


School 1 23.72 (0.83) 23.5 (23, 24) 
School 2 19.48 (1.17 

School 3 19.14 (1.01) 19 (18, 20) 
School 4 23.72 (0.67) 24 (23, 24) 


A pre-specified FSM subgroup analysis was conducted. The effect of the intervention on pupils 
eligible for FSM was assessed through the repetition of the primary analysis for this subgroup and 
through inclusion of an interaction between FSM status and group for each school. The subgroup 
analyses showed no evidence of the intervention differentially benefiting pupils eligible for free school 
meals in any of the schools. The pooled estimate for the subgroup analysis was 0.85 (-1.51 to 3.21) 
and effect size was 0.08 (-0.13 to 0.30). 


The exploratory analysis of the pupils in the mini RCT (between the cut points) was conducted. The 
mean PIM 12 score for the pupils in the intervention was 25.6 (SD 7.7) and for control pupils was 24.3 
(SD 6.5). The effect size was very small (0.02 95% Cl -0.43 to 0.47) and there was little or no 
evidence of a difference between the two groups (0.19 95% Cl -3.56 to 3.94, p=0.88). 


Cost 


The funding provided to each school to employ staff and run the programme was £52,000. If the 
intervention took place with 60 pupils as recommended, this is equivalent to around £870 per pupil 
per year. 


Schools in the evaluation did not pay anything towards the intervention. The ‘Making it Work’ 
handbook states that projects are normally up to 100% funded for the first three years while for Years 
4 to 6 schools would be expected to fund or find funding for at least 50% of thecosts with this rising to 
at least 70% of costs in Years 7 to 9. 


Education Endowment Foundation 22 


SHINE in Secondaries 


Process evaluation 


This report presents a cross-sectional snapshot of two site visits to SHINE on Secondaries schools 
(School 1 and School 3), conducted near the end of the year. It discusses positive features of the 
delivery of the intervention and some barriers to its successful implementation. The analysis is based 
on data collected through teacher interviews in School 1 (one interview with the project manager and 
one interview with the head of maths) and School 3 (one focus group interview with the two project 
managers, four teachers and two peer mentors), a focus group interview with five pupils in School 1 
and lesson observations in both schools (six lessons observed overall) carried out by the process 
evaluator. Quality assurance of the observations for this evaluation was done during an observation 
of the Hallé SHINE on Manchester intervention where the same researcher conducted a joint visit with 
the principal investigator conducting similar observations. They reached high agreement in their 
observations during this first visit. 


An interview with the SHINE programme manager and the SHINE chief executive conducted by two 
project researchers at the end of the programme also detailed their thoughts on implementation, what 
went well and future directions. 


Excellent features of the delivery 
Teacher perceptions 


There was consensus among the teaching staff in School 3 that one of the most important factors for 
the successful delivery of the intervention was the good work of the two SHINE project managers who 
were also teachers in the school. This included the organisation of Friday school assemblies 
preparing pupils for their SHINE day and regular parent events which celebrated the work that was 
produced. The Friday school assemblies, in particular, seemed to have been initiated by this 
particular school as this event was not mentioned by School 1. All eight teachers interviewed in the 
two schools agreed that SHINE in Secondaries gave them the opportunity to think and plan ‘outside of 
the box’ and they enjoyed teaching around a particular theme, which helped reinforce links with ‘real 
life’. 


It was also felt that the continuity with the school week that the SHINE in Secondaries programme 
provided for pupils in Year 7 was an excellent feature of the intervention. From our visits, it seemed 
that all SHINE in Secondaries teachers see all or at least some of the pupils during the week. This 
allowed the teaching content of the SHINE in Secondaries programme to better target pupil needs 
and also to build on the pupils’ familiarity with the teaching staff. In School 3, for example, the SHINE 
teaching content was decided following a consultation with the literacy and maths school teachers 
about what particular knowledge and skills they would need to target next. The English teacher, in 
particular, identified pupil needs during the week and targeted those areas on Saturdays. In School 1, 
the head of maths mentioned that the relationships between the pupils and the teachers were 
stronger because of the good continuity between the children’s school week and SHINE. The pupils 
felt more comfortable to approach and interact with the teachers in school. She also felt that there 
was an enhanced sense of identity among the pupils, a sense of belonging and better relationships at 
all levels. Pupils were given the space and the time for more individual guidance and support which 
was not often possible to provide in school. The head of maths also mentioned that some pupils had 
achieved or exceeded their targets in school because of this extra support in SHINE. 


All teachers interviewed agreed that pupils liked the familiarity and the relaxing atmosphere that 
SHINE in Secondaries encouraged and that they learned while being engaged in an enjoyable way. 
An important outcome of the intervention reported by the teachers in both schools was pupils’ 
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improved social skills, better relationships at all levels and increased confidence through their 
participation in practical and team-building activities. 


Pupil perceptions 


The five pupils who were interviewed (in School 1) spoke enthusiastically about their SH/NE in 
Secondaries experiences. All pupils mentioned that they had made good progress in their learning 
while having fun. Some pupils provided specific examples to illustrate their learning in maths or 
English, as the following examples illustrate: 


‘It makes it easier to get high levels in maths. ...We have learnt about fractions.’ (boy) 
‘In English we have done colons and semi-colons, punctuation, spellings.’ (boy) 


‘In English we learnt about Shakespeare, when he was born and when he died, what plays he wrote, 
the people he met etc.’ (boy) 


Pupils also talked with enthusiasm about some of the trips they had been to as part of SHINE in 
Secondaries, not least because of the learning and the links with subjects such as English and 
science that they could make. Furthermore, these pupils enjoyed making new friends. Some of them 
were in the process of applying to be a student mentor the following year as they felt they would enjoy 
taking on a more senior role with the new cohort of SHINE in Secondaries pupils. 


Lesson observations 


The six lessons observed in the two schools demonstrated a number of creative teaching ideas that 
were used effectively to provide variety, interest and engagement. All lessons observed were 
planned thoroughly and delivered engagingly. Questioning was often used effectively and teacher 
guidance and feedback during group work was ongoing. Overall, pupils were encouraged and helped 
to exercise initiative, as well as an element of choice and independent thinking, in the lessons 
observed. These teaching and learning approaches are well aligned to the guiding principles for the 
SHINE in Secondaries curriculum as outlined in the ‘Making it Work’ handbook. Teaching assistants 
and peer mentors were used effectively in all of the lessons we observed. For example, in the literacy 
lesson observed in School 3, the teaching assistant and a Year 10 peer mentor offered support to 
pupils during their small group work. The peer mentor worked mainly with one group whereas the 
teaching assistant moved from group to group offering pupils advice with their work. 


All pupils appeared attentive and on task, thus demonstrating positive and constructive relationships 
with the teacher and each other. In School 3, for example, all pupils were engaged and on task 
throughout the literacy session observed. Through a series of practical tasks and effective 
questioning, the pupils were reminded of important grammatical rules which they were then 
encouraged to use in their own writing. Pupils’ behaviour was excellent and the learning environment 
was comfortable, enjoyable but appropriately challenging. The teacher was encouraging but did not 
readily accept pupils’ answers. Instead, she encouraged deeper thinking and more elaborate 
answers. Similarly, in the numeracy lesson observed in School 3, the teacher explained engagingly 
the value of designing a questionnaire, guided pupils through the process of designing good 
questions using a series of interactive activities and provided constructive feedback to pupils 
throughout the session. Pupils were encouraged to think for themselves and were given opportunities 
to receive feedback and improve their work. The learning environment was particularly comfortable 
mainly due to the warm and engaging personality of the teacher. 
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Barriers to successful implementation—suggestions for future development 


Most lessons adhered to the guiding principles of the SHINE in Secondaries curriculum. However, in 
a small number of the six lessons observed, we found that pupils could have been more cognitively 
challenged. 


Both schools identified pupil attendance and the initial pupil recruitment process as the two biggest 
barriers to the successful implementation of the intervention. Pupil attendance was a continuing 
concern but both schools were particularly proactive, offering instant rewards, certificates for 
participation and liaising with the parents on a regular basis. The challenge in terms of time required 
to attend SHINE in Secondaries on a Saturday was also identified by some pupils interviewed. 
Finally, teachers in School 3 expressed their concern that the initial recruitment process excluded a 
number of pupils who would have been particularly keen to attend. 


When SHINE is run without the evaluation/trial aspect, schools would be given guidance from SHINE 
on how to select pupils. The selection criteria are that pupils experience an indicator of disadvantage 
(e.g. FSM, EAL, SEN or other indicator as known by the school) or that they are underachieving (this 
can be broadly defined—low achievers as well as those in the middle of the field who teachers think 
are capable of more could be included). The schools select the pupils but do need to report to SHINE 
on why pupils are deemed eligible to attend. The evaluation lessened the scope for the pupils who 
could attend and limited the school from inviting additional pupils through the year if spaces became 
available. It is anticipated that if this intervention was rolled out there would be more scope for 
teachers to select a wider variety of eligible pupils but that the issues with attendance during the 
evaluation would not be eliminated. 


Feedback from the developer 


The developer felt that the implementation of the SHINE in Secondaries programme had gone well in 
schools and that schools were following the model with fidelity. Schools had reported to them that the 
programme had had a positive effect on pupils’ confidence and self-esteem at a particularly 
vulnerable time of transition to a new school. It also gave the teachers in the school the opportunity to 
get to know the pupils better in a safe environment and this also helped them teach better during the 
week. Teachers felt that the key to the programme being successful was that it was something 
different from school during the week and that it wasn’t just individual subject catch-up. The 
programme was seen by the teachers as an opportunity to instil a ‘love for learning’ in pupils through 
the extra-curricular and ‘real-life’ experiences. 


The SHINE team did mention some difficulties that had been faced during the year. The first 
challenge was with continuity of the programme when leadership staff involved in planning moved on 
to different roles or schools. The second was with maintaining good attendance from pupils through 
the course of the school year, particularly with the evaluation restricting who could be invited to attend 
(as described above). Reasons for not being able to attend the programme were other Saturday 
commitments, for example sporting commitments, religious school, or difficult family circumstances 
(e.g. single parent families, or living with extended family). There seemed to be good parental 
engagement in both schools observed. In School 3, for example, the children’s work was celebrated 
in regular parent evenings. Both schools contacted the parents every week, if needed, to encourage 
pupil attendance. Going forward, SHINE would be keen to try and increase the amount of parental 
engagement with the programme and look at ways to encourage better attendance. Examples of 
good practice in schools had been when the school had incorporated the SHINE in Secondaries 
programme into the normal school week with pupils who attended wearing badges, a good launch 
event for all eligible pupils and teachers following up on SHINE work with pupils during the week. All 
of these things were thought to promote attendance on a Saturday. 


Education Endowment Foundation 25 


SHINE in Secondaries 


Conclusion 


Key conclusions 


1. Impact of the intervention: Very small effect sizes on the primary and secondary outcomes of 
literacy and numeracy were detected. As this was an exploratory evaluation design and the 
number of schools involved in the project was small. This made it unlikely that any impact of 
SHINE could be detected, should the true estimate of effect be small. 


2. Implementation and organisation issues: Feedback from the teachers and pupils suggested 
that the intervention was perceived to be valuable in providing an opportunity to consolidate 
literacy and numeracy skills. 


3. Feasibility of evaluation design: The purpose in piloting this design was to explore the 
feasibility of using this approach when randonlly allocating pupils to an intervention group was 
not possible or desirable. The regression discontinuity design is an appropriate evaluation 
method for education interventions. However, because it is less powerful than a randomised 
controlled trial, the number of pupils involved must be two and a half times greater if a small 
effect is to be detected. 


In this evaluation we have undertaken a prospective RD evaluation. We have also successfully 
demonstrated the feasibility of nesting a tie-breaker RCT within a RD study. We are unaware of any 
study conducted in the UK that has used this novel approach. Because we have demonstrated its 
feasibility we can recommend its use in other evaluation situations where randomisation is either not 
possible or not feasible. Note, however, that a ‘standard’ RCT will be the superior design in most 
instances. 


The impact evaluation using RD design did not find evidence of a statistically significant impact of the 
intervention on any of the outcomes. However, this finding should be treated with extreme caution, 
given that this was a pilot evaluation and underpowered to detect significant small effects (Should they 
exist). 


Given the funder provided only four schools as the sample size at the outset, we only had the power 
(80%) to detect (as statistically significant) relatively large effects (0.5). 


When an RCT is not possible the RD design is the strongest quasi-experimental design providing a 
robust estimate of effect as long as the sample size is large enough (at least 2.75 times the size for a 
RCT). This pilot evaluation is an important step in establishing evidence of promise of the impact of 
SHINE in Secondaries, and equally important for exploring the feasibility of using the RD design. 


In this evaluation the observed effect size was not statistically significant; however, this does not 
invalidate the use of the design. A key disadvantage of the RDD is that it is heavily dependent on 
specifying the correct functional form and if this is incorrect then it can lead to biased estimates—this 
highlights the importance of undertaking extensive sensitivity checks which are sufficient to check the 
robustness of the findings. In this analysis we chose to undertake a parametric approach rather than 
a non-parametric approach. The parametric approach uses all of the data to model the outcome and 
uses different functional forms for the rating variable. The non-parametric approach limits the 
analysis to data that lie within a window (bandwidth) where the functional form is more likely to be 
linear. The main difficulty with this approach is the selection of the optimum bandwidth. Given the 
relatively small sample size and need to analyse the schools individually in this pilot, the parametric 
approach seemed to be the most appropriate. In this analysis a regression framework was used. It is 
possible that other analytical approaches may have led to different estimates of the treatment effect, 
for example using risk-based allocation methods (Finkelstein et a/., 1996), non-parametric regression 
(Hahn et al., 2001) or local linear regression (Imbens and Lemieux, 2007). 
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Different cut points were used in the two larger and two smaller schools and this had an impact on the 
analysis. In order to adequately capture the clustered nature of the data the analysis was undertaken 
for each school and then pooled in a random effects meta-analysis. If there were a larger number of 
schools in the study, with two different cut points, then it might have been possible to analyse the 
schools together using the same cut point and account for the clustered nature of the data as a 
random effect, and then pool the two estimates in a meta-analysis. This would obviously increase the 
available data and might lead to better specification of the functional form. In this analysis we could 
have combined the two larger schools and two smaller schools and undertaken the analysis on each 
set, but the only way to account for the clustering would have been as a fixed effect which might not 
have been adequate. 


The process evaluation was ‘light touch’ and therefore any findings should be treated with extreme 
caution. The process evaluation found many positive features and perceptions of the intervention 
from the perspective of the various stakeholders, including pupils. 


The RD experimental design meant that pupils were selected to attend the intervention based on a 
score on an English assessment. Teachers would normally have more discretion regarding who to 
invite and which pupils would benefit (as long as they met one of the SHINE-specified criteria). Some 
pupils who were eligible for the intervention already had other Saturday commitments such as 
religious school or sports programmes and therefore would not have been able to attend had they 
been invited. 


Loss to follow-up was higher for the intervention group than for the control group which may reflect 
attendance rates at school, with those who performed better at the beginning of the year having 
higher attendance generally. 


Uptake of places at the intervention was 74% of those who were assigned to the intervention group, 
which means that 26% of those counted in the ITT analysis did not attend the intervention. The ITT 
analysis provides average impacts of the intervention for those who were offered the intervention, 
whether or not they actually participated in the intervention. This created a ‘fuzzy’ regression 
discontinuity and reduces the robustness of the evaluation. The attendance also meant that the ratio 
of staff to pupil was higher than it would normally be. However, the dilution effect of the ITT estimate 
in the presence of non-attendance is similar for the RDD as for an RCT. However, average individual 
attendance of those children who accepted places was 68%. This includes any pupil that is 
considered as taking up their place by attending a minimum of one session. The average attendance 
is based on the numbers of sessions possible in each school. 


Nevertheless, this pilot demonstrated that designing and implementing an impact evaluation using RD 
design is feasible, and this was a successful outcome of the evaluation. 
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Secondaries planning document 


Lunch 


18.1.14 challenge 


Learning 


25.1.14 | valk 


1.2.14 


OVERALL FOCUS: What makes a healthy child? (Exploring healthy lifestyles) 


Literacy |Basic breakdown 
Date Numeracy focuses foriens of the day 


Ingredients and budgets — 


-Calculating costs (using rounding to estimate 
costs and the use of repeated addition vs 
multiplication) 


-Using a budget (tracking expenditures by 
creating formulae on excel) 


-Best Buy options (discussion using ratio to 
determine best value, i.e. multi-packs vs 
singles) 


-Understanding nutritional information 


Route planning — 


-Grid reference (understanding coordinate and 
other map referencing systems) 


-Map reading (discussing keys/symbols and 
scale notations such as 1:500) 


-Journey statistics (calculating 
speed/distance/time and representing 
movement on a distance/time graph) 


-Calories burnt, steps per minute required. 


Managing budgets — working out a weekly 
budget for their family (money often being a 
cause of stress) 


-Calculating costs (using rounding to estimate 
costs and the use of repeated addition vs 
multiplication) 


-Use of ICT (tracking expenditures by creating 
formulae on excel) 
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Example of a medium-term SHINE in 


Communication 
skills — using a 
shopping list, trip to 
supermarket 


Creating the sandwich 
— awareness of food 
hygiene and 
cleanliness 


Persuasive skills— | _ 
Visit to supermarket 


Making the sandwich 


Persuasive skills — 
Dragon’s Den style 


: Dragon’s Den 
presentation 


Communication 
skills — 
geographical 


Visitor centre 


Botanical gardens 


Discussion 


discussion 
activities — what 


Freedom to express 
opinions 


friendships make 


you feel? Peer pressure 


Health 


Presentational 
features — create 
posters advising 


Staying safe 


30 
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following the plot |Cinema experience 
; of the film, 
8.2.14 Oe understanding how |Creative dance 
it relates to current 
news 


Cultural celebration of 


-Info graphics (where do different foods come |Communication diferent toads? 


from, most popular, most healthy) skills — creating 
stalls including the 


-Costs of different food different cultures Advertising — creating 
Celebrating for their culture 


different : ; : ; : 
cultures |-Calorie counting (making comparisons Understanding the 


between the DRA for men/women) SOONG C-SeBIg Ue 


stalls 


-An introduction to low/high GI foods (base 


100) vs calorie counting pies weloMy Seung 


of food 
Using numeracy in sports — | 


-Averages (mean, median and mode. The 
advantages of choosing the right average) 


: : : Visit to National 
Experience |-Range (a measure of consistency of d Cycling Centre to 
sport performance) 2 : watch BMX practice 
sessions 11-1pm. 


-Info graphics (representing results/ winners 
etc) 


-Calculating calories burnt from exercise 


‘Experience sports 


Competitive maths — 
Sports reporting — } 
-Calculating lap times understanding . Table tennis 
(minutes/seconds/milliseconds) what you need in Sport Relief 
terms of stats, mile 


. good questions to . Reaction test 
-Averages (mean, median and mode. The ask competitors 


advantages of choosing the right average) etc Speedway will only be 


attended by the press 
-Range (a measure of consistency of Conducting team 


performance) interviews with the 


winners, writing up 
-Comparisons (speed, comparing personal reports 
times with Olympic/World records and peers) 


Developing an understanding of health advice 


through statistics. 
Carousels 


Using a range of statistical diagrams to back 


evidence/arguments (including Info graphics) aoa for parents / 
awar 
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Appendix 2: Schools information sheet 
SHINE in Secondaries 


Evaluation Information Sheet 


The Education Endowment Foundation has commissioned Durham University to evaluate 
the SHINE in Secondaries programme. This evaluation aims to find out how well the 
programme works in improving educational outcomes for the children involved. The 
research team, led by Professor Carole Torgerson, have designed a study that will assess 
gains in literacy, numeracy and attitudes towards learning for the children who take part in 
the programme compared to the children who do not. The evaluation will involve schools 
assessing all Year 7 pupils at the beginning and end of the school year. The assessment at 
the beginning of the school year will be used to determine which pupils are eligible to 
attend the SHINE in Secondaries programme. 


Assessment in September 2013 

At the beginning of the school year, we ask that all Year 7 pupils complete the Progress in 
English assessment which will be administered and marked by Durham University. We will 
arrange a Suitable time with schools for this to take place in the first half of September. This 
test should be conducted under exam conditions and should take around an hour. 

Durham University will mark the assessments and then provide schools with a list of 
eligible pupils to invite to attend the SHINE in Secondaries programme at the end of 
September. We will also provide schools with additional pupils to invite if there are still 
places available before the programme starts. Schools will also receive the data from the 
assessment. 


How Durham decide on eligible pupils 

All pupils who score below a pre-determined cut-off point in the test (set by Durham 
University) will be eligible to attend the intervention. Durham University will provide a list 
to schools of pupils that they should invite to attend SHINE in Secondaries. It is important 
that all of these pupils are invited, even if you strongly believe they will not attend. Those 
pupils that score above the cut-off point are not automatically eligible for the SHINE 
programme. Durham University will try to ensure that the cut-off point is set so that around 
60 pupils will end up attending the programme. Pupils that score just above the cut-off 
point (and below a second higher cut-off point) will make up a random ordered waitlist of 
pupils. We will ask schools to let us know how many pupils have accepted their place on 
16th October. If there are still spaces available before the programme starts, Durham 
University will issue you with a list of additional pupils to invite from the waiting list. 


Durham University’s involvement through the year 

Schools will run the SHINE in Secondaries programme from November 2013 to July 2014. 
During this time the Durham University researchers will observe some sessions and talk to 
stakeholders including children, parents and staff regarding the programme to learn about 
how the programme is perceived and whether any improvements could be made. 


Outcome assessment in June/July 2014 

At the end of the school year we ask that all Year 7 pupils sit the Progress in English 
assessment again as well as the Progress in Maths assessment. We will also ask pupils to 
complete a short questionnaire looking at attitudes to school and to learning. As before, 
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these assessments will be administered by a researcher from Durham University on a 
suitable date arranged with the school, and will take approximately 2 hours. These will 
again be done under exam conditions. 

The assessments will be marked by Durham University and the results will be used to 
assess the impact of the programme on literacy and numeracy. We will provide schools with 
your individual pupils marks in both the English and Maths assessments at the end of July 
2014, which we hope will be of use to you. 


Use of data 

All pupil data will be treated with the strictest confidence. No individual school or child will 
be identified in any report arising from the research. Named data will be matched with the 
National Pupil Database to enable longer term follow-up and we will also share the data 
with SHINE, with EEF data managers and with the UK Data Archive for research purposes. 
Durham University will provide documentation for schools to inform parents about the 
assessments in September and June/July. Parents will be able to opt out of their child’s data 
being used in the evaluation by informing the school who will pass this on to the project. 


Summary 

To meet the requirements of the evaluation we would like schools to agree to: 

e Durham University researchers coming into the school to assess all the Year 7 pupils in 
September 2013 and in June/July 2014 on dates agreed mutually. 

e Invite all eligible pupils (as identified by Durham) to attend the SHINE in Secondaries 
programme. Pupils on the waitlist can only be invited if there are spaces and should be 
invited as directed by Durham University. 

e Allow Durham University researchers to visit the SHINE programme at agreed points 


through the year. 

e Provide Durham University researchers with unique pupil numbers (UPNs) for all Year 
7 pupils. 

Timeline What happens Date 

Progress in English assessment completed | September 2013 

by all pupils 


Schools given list of eligible pupils to invite | 30th September 2013 
to SHINE in Secondaries 


Schools report on number of pupils who 16th October 2013 

have accepted place 

Durham issue list of additional pupils to 21st October 2013 

invite if necessary 

SHINE in Secondaries runs November 2013 - June 2014 


Progress in English and Progress in Maths | June/July 2014 
assessments completed by all pupils 


Education Endowment Foundation 33 


SHINE in Secondaries 


Appendix 3: Schools consent form 


Independent Evaluation of SHINE in Secondaries 
School consent and registration form 


Please complete the details below and return the form to Kirsty Younger (contact details 
overleaf). This will allow us to register your school for the evaluation and to contact you by 
email or post about the evaluation. 


SHINE ProjeGt: Managers ccccceceviestcesoecapacceiesteve sen cecciedecersienssias'gesaiiaceiiect acest sateeedecuedseastiaivasdiedzigectsiereesatenss 
Tel No (if different from abOVE) 1 :...:ccc.ccccccseecescedadesccececesiaceeceetsebecceteceeeaeeoiencdeeeesesetadeneesasgbenserenneenaeesene 


FTW a oa iid baact iad caadesbadiatia doe coivendeadhieacdiadia aden ad aaeadiaanbeeostad ee nadhaua adv Gstasvanuadnina arena a uuuaa eens 


Please give details of anyone else at the school that should be on our circulation list 


Number of pupils for 2013/14 in Year 7: .....e eee 
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Please read the information below carefully. It is important that you understand the 
requirements of the evaluation. Please ask us if you have any questions. 


e [confirm that I have read and understood the information sheet for the SHINE in 
Secondaries evaluation and have had the opportunity to ask questions. 


e J understand that pupil assessment data will be collected using Progress in English and 
Progress in Maths assessments; the assessments will be administered and marked by the 
evaluators at Durham University. I understand that named data will be matched with the 
National Pupil Database and shared with EEF and the UK Data Archive for research 
purposes. 


e [understand that all pupil data will be kept confidential and that no material which could 
identify individual children or the school will be used in any reports of this evaluation. 


e J agree to Durham University researchers coming into the school to assess all the Year 7 
pupils in September 2013 and in June or July 2014 on dates agreed mutually. 


e J understand that all eligible pupils (as specified by Durham) should be invited to attend the 
SHINE in Secondaries programme as directed by Durham University and that pupils who are 
not eligible should not be invited to attend the programme. 


e [consent on behalf of the school to take part in the above study. 


Signatune;:of headtEacher ie.csccso cece cccectietsadiveseet pecteweeheetdevree qeetede tre dtie ve sliaada alate 


Thank you for agreeing to take part in this important research. 


Please return this consent form by post, or scan and email to: 

Kirsty Younger, Research Assistant 

kirsty.younger@cem.dur.ac.uk 

0191 334 4176 

CEM, Rowan House, Mountjoy Research Centre, Durham University, Stockton Road, Durham 
DH1 3UZ 
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Appendix 4: Parent opt-out letter 


School of Education 
Durham University 
Leazes Road 
Durham 

DH11TA 


<insert date> 


Dear Parent or Guardian 


Evaluation of SHINE in Secondaries Saturday School programme 
I am writing to let you know about a study which will be taking place in your child’s school this year 
(2013/2014). The Education Endowment Foundation (EEF) is funding the study to find out how much 
the SHINE in Secondaries programme improves the maths and English skills of pupils in Year 7. The 
study is being carried out by Durham University. 


The headteacher of your child’s school has given permission for the school to take part, and the study 
will involve all pupils in Year 7. In September, pupils will complete a literacy assessment. The results 
of this will be used to establish which pupils might benefit most from attending the SHINE in 
Secondaries programme. These pupils will be invited by the school to attend the programme. At the 
end of the school year, all Year 7 pupils will complete a maths and literacy assessment. The school 
will receive the results from all the assessments to inform teaching. We will also ask pupils to 
complete a questionnaire about their attitudes to school and learning. We will use the results of the 
assessments to see how much the SHINE in Secondaries programme helps improve educational 
outcomes. 


The school will also pass on some background information about pupils (date of birth, gender, and 
Unique Pupil Number) to Durham University to help evaluate the programme. All data collected will 
be treated confidentially and will only be used for research purposes. The data for your child’s school 
will be analysed anonymously, together with data from other schools, and no individual pupils or 
schools will be named in any report. Pupil assessment data will be matched with the National Pupil 
Database for longer term follow-up and also shared with EEF data managers, and the UK Data 
Archive for research purposes only. 


I do hope that we have your support for this important study and that you are happy for your child’s 
data to be used in this way. If you would prefer we did not use your child’s data in this research study 
please contact the school to opt out at any time during the project. 


If you have any questions about the study please contact Kirsty Younger, Research Assistant on 0191 
334 4176, or email kirsty.younger@cem.dur.ac.uk. 


Yours faithfully 


(alte we Tor ech 


Carole Torgerson 


Principal Investigator 
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Appendix 5: Opt-in parental consent form for focus group 


School of Education 
Durham University 
Leazes Road 
Durham 

DH11TA 


19" june 2014 


SHINE in Secondaries Evaluation by Durham University 
Dear Parent/Carer 


We are writing to the parents/carers of all pupils attending the SHINE in Secondaries Saturday School at 

. Durham University has been funded by the Education Endowment Foundation to evaluate the 
SHINE programme and to look at any improvements that could be made in the future. As part of our evaluation, 
researchers from the university will be visiting the project between now and July to observe a session and talk to 
the children attending and the staff running the programme. 


We would like to hold a focus group with some of the participating pupils and would like your consent for your 
child to be included. The focus group will involve one or two researchers from the university meeting with a small 
group of pupils (4-5 children) and member of staff from the project. We will ask the children questions about their 
experience of attending SHINE. Participation in the focus group is voluntary and no individuals will be named in 
any report about the project. We expect the focus group to last for about 25 minutes and to take place during the 
SHINE session on 5" July. 


If you are happy for your child to be involved in a focus group please complete the tear-off sheet below and return 
it to the SHINE in Secondaries project next Saturday. 


If you need further information about the evaluation or have any questions please contact the Evaluation Team 
through Clare Collyer, Project Administrator: clare.collyer@cem.dur.ac.uk; 0191 334 4682. 


Yours faithfully 


Ute J. Torna. 


ee 
Professor Carole Torgerson 


Principal Investigator 


| give consent for my child to be included in a focus group for the SHINE in Secondaries evaluation to 
find out about the pupils’ experiences. 


SRN CES: AUN Geiss igse di iearcehiddaventvitenen dh vanigiceieeh sida reat Oban Gotamacd oa ude bunaueuibeeasieetelnine dada deabadeua Mewes 
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You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/2 
or email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission 
from the copyright holders concerned. The views expressed in this report are the authors’ and do not 
necessarily reflect those of the Department for Education. 


This document is available for download at www.educationendowmentfoundation.org.uk 


A Education 


Endowment 
Foundation 


The Education Endowment Foundation 
9th Floor, Millbank Tower 

21-24 Millbank 

London 

SW1P 4QP 


www.educationendowmentfoundation.org.uk 


